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Abstract 

Agricultural education has been criticized for publishing research lacking many of the rigorous 
qualities found in publications of other disciplines. A few agricultural education researchers 
have suggested strategies for improving the rigor with which agricultural education studies 
report on methods and findings. The purpose of this study was to determine whether studies in 
agricultural education have followed the methodological suggestions proposed by these select 
researchers. Every study published in the Journal of Agricultural Education between 2000 and 
2006 (N = 244) was examined using content analysis techniques. Specifically, quantitative, 
inferen tial studies were assessed with regard to the conveyance of practical significance, the 
stating of research hypotheses, and explicitly acknowledging how statistical assumptions were 
addressed and confirmed. Of the 88 inferential studies published in the Journal since 2000, 52% 
did not convey the practical significance of their research findings and 74% did not explicitly 
state research hypotheses. Nearly 80% did not acknowledge how the assumptions associated 
with statistical tests used were addressed and confirmed. Studies that conveyed practical 
significance, stated hypotheses, and addressed statistical assumptions are cited as examples to 
follow. Perhaps an increase in pages allotted to each article would provide authors with 
adequate space to report such items. 


Introduction/Literature Review 

The character of a room can be judged 
by the layout of the pictures on its walls. 
Similarly, the character of a discipline can 
be judged by the layout of its research 
(Williams, 1991). The discipline of 
agricultural education has been criticized for 
publishing research which lacks many of the 
rigorous qualities found in the publications 
of other disciplines (Buriak & Shinn, 1989; 
Dyer, Haase-Wittler, & Washburn, 2003). 
Considering each picture on the wall of a 
room analogous with an individual research 
study might provide some guidance on what 
our discipline could do to improve its image 
among similar fields. Although the content 
of the pictures may vary widely, just as the 
content of research studies in our discipline 
varies, the steps each photographer took 
from snapshot to photograph development 
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should be quite similar. In this sense, 
researchers are much like photographers, 
taking snapshots of phenomena and 
following a series of iterative steps to 
“develop” their ideas into pieces of 
knowledge that add character to a discipline. 

From a methodological standpoint, an 
understanding of the consistency with which 
such steps have been followed may provide 
insight as to why agricultural education 
research has been viewed as less rigorous 
than other fields. In fact, some researchers in 
the discipline (most notably Miller, 1994, 
1998, 2006) have strongly advocated that 
researchers follow specific guidelines during 
the measurement, analysis, and reporting 
phases of their studies. In a sense, Miller 
(1994, 1998, 2006) advocated that the steps 
taken to “hang our pictures” are consistent 
across studies. However, have studies in 
agricultural education followed the 
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methodological suggestions proposed by 
Miller and others? This paper will attempt to 
answer that question via a “snapshot” of 
research published in the Journal of 
Agricultural Education since 2000. 

Wagner (1993) defined research as a 
strategy for reducing ignorance and 
advocated that educational research 
reporting is a form of teaching. As such, in 
the same way agricultural education 
instructors discuss the theoretical and 
pedagogical assumptions grounding their 
classroom lessons, it seems that the way in 
which we report our findings should mirror 
the format of our lesson plans. In a sense, if 
research is to be replicated, one must know 
relevant background information, objectives 
to be achieved, materials needed, and the 
specific procedures (methods) used to 
accomplish established objectives. However, 
have we become so excited to report our 
findings (or jump to the evaluation exercise 
in our lesson plan) that we forget to also 
teach our research methods (or let our 
learners experience lesson activities) to the 
reader? 

It is important to consider the clarity 
with which we report our research findings, 
but others advocate that the clarity and 
consistency with which our research 
methods are reported is just as important. 
Referring back to the lesson plan analogy, 
how could a teacher replicate a lesson 
without step-by-step instructions detailing 
the procedures to use? In fact, Miller (1994, 
1998, 2006) has suggested for quite some 
time that research in the agricultural 
education discipline lacks clear, rigorous 
documentation of the measurement and 
analysis procedures used to arrive at the 
reported findings — from research 
hypotheses to practical significance. For 
example, Miller (1994, 1998) suggested that 
researchers in agricultural education do not 
explicitly (in writing) state how their 
analysis process initiated, specifically (a) the 
null and alternative hypotheses and (b) how 
assumptions associated with the statistical 
tests used during the analysis were 
addressed and confirmed. 

In essence, if one was to develop a 
lesson on how to properly hang a picture on 
a wall, his/her lesson plan would surely 
contain (a) a description of the process to 
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determine where to hang the picture 
(hypotheses and assumptions) and (b) how 
to determine that the location chosen will 
support the weight of the picture and frame 
(checking and confirming assumptions). 
This might include a description of the 
“tools” used to check these assumptions 
(such as a stud finder, hammer, and level), 
just like Levene’s test is used to ensure 
homogeneity of variance in a regression 
analysis. In addition, Miller (1994) 
suggested that although studies in 
agricultural education are using inferential 
statistics, they are not addressing the 
practical (compared to statistical) 
significance of their research findings and 
that doing so will improve the utility of the 
discipline’s research findings. This would 
imply that for every inferential statistical 
procedure used, the percentage of variability 
explained should be reported. 

Others echo the importance of 
addressing the practical significance of 
research findings when inferential statistics 
are used. Williams (1991) suggested that the 
discipline’s research reporting techniques 
must have applied applications so as to help 
promote partnerships with agencies and 
industry and potentially facilitate 
dissemination and implementation of 
research findings. Specifically, Miller 
(1994) suggested squaring the Pearson 
correlation coefficient to get at practical 
significance. Agresti (1996) advised that 
social science researchers analyzing 
categorical data compare odds ratios to 
determine practical significance. Agresti and 
Finlay (1997) suggest that social science 
researchers use confidence intervals more 
often to convey the practical significance of 
research findings. For example, instead of 
stating whether there was a significant 
difference in the mean amount of time that 
males and females watched television (as 
could be indicated with an independent 
samples /-test), they advocate reporting how 
different a similar population would likely 
be in their mean television watching time 
with a confidence interval. Miller (1998) 
suggested that researchers address practical 
significance by calculating and reporting 
omega-squared or eta-squared coefficients. 
Finally, Ary, Jacobs, Razavieh, and 
Sorensen (2006) suggest reporting effect 
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size to assess the magnitude of difference 
between two or more groups and convey 
practical significance to the research 
consumer. That is, in the same way teachers 
answer the “so what?” question at the 
conclusion of a lesson plan with 
recommended application/ extension 
activities, researchers should answer the 
practical significance question of their study 
findings. 

Miller (1998) spoke specifically with 
regard to data analysis procedures in 
conveying the practical significance of 
agricultural education research results to the 
research consumer. He advocated that 
although complicated statistical procedures 
often seem to be preferred in agricultural 
education research, such procedures “do not 
make something scientific or important” 
(Miller, 1998, p. 8). The practical 
significance of the questions researchers ask 
must not be masked by “glamorous” 
analysis procedures. More recently, Miller 
(2006) spoke to the practical ability of our 
research findings, stating “researchers in 
agricultural education must become better 
able to articulate their possible contributions 
to the research missions of a college and 
communicate clearly and persuasively” 
(P-114). 

Miller (1998) suggested that the Journal 
be used as a forum to discuss issues such as 
determining the practical significance of 
research findings and ensuring that data 
analysis procedures are appropriately 
applied, given research hypotheses and 
statistical assumptions (e.g., nonnality, 
linearity, homoscedasticity). As the premier 
refereed publication in agricultural 
education, the authors wished to determine 
the extent to which studies in the Journal 
have followed the suggestions that Miller 
(1994, 1998, 2006) and others have been 
making for over a decade. 

Purpose and Research Questions 

The purpose of this study was to 
determine the rigor in agricultural education 
research reporting in the Journal since 2000. 
Given the suggestions proposed by Miller 
(1994, 1998, 2006) and others, the following 
four questions guided this work: 


1. What type of articles and how many 
of each type have been published in 
the Journal since 2000? 

2. To what extent have quantitative 
studies published in the Journal that 
use inferential statistics conveyed the 
practical significance of their 
research findings? 

3. When necessary, to what extent have 
such studies explicitly stated 
research hypotheses? 

4. To what extent have such studies 
explicitly stated how the assumptions 
associated with statistical tests were 
addressed and confirmed? 

Methods and Procedures 

To measure whether studies in 
agricultural education have followed the 
suggestions proposed by Miller and others, 
and continuing with the lesson plan analogy, 
the lead author developed a rubric to score 
each study reviewed. The rubric allowed the 
author to compare the type of article 
published with whether (yes/no) and to what 
extent practical significance, research 
hypotheses, and statistical assumptions were 
conveyed. Every study published in the 
Journal between 2000 and 2006 (. N = 244, 
not including seven distinguished lectures) 
was examined using content analysis 
techniques in an attempt to answer the first 
research question. However, studies using 
quantitative methods were more thoroughly 
examined in an attempt to answer Research 
Questions 2, 3, and 4. Only the lead author 
reviewed all studies to enhance the 
reliability of how each article was evaluated 
and eliminate inter-rater reliability issues. 
When reviewing studies, “rigor” in terms of 
research reporting was operationally defined 
as the ability of an article to appropriately 
convey practical significance, research 
hypotheses, and associated statistical 
assumptions. 

With more than 83% of the studies 
published in the Journal between 1990 and 
1999 being classified as quantitative in 
nature (Dyer et ah, 2003), the authors 
posited that the majority of studies published 
between 2000 and 2006 would also be 
quantitatively based. Studies using mixed 
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methods (qualitative and quantitative 
techniques) were examined with the same 
rigor as those using only quantitative 
methods because both descriptive and 
inferential statistics can be used in such 
studies. Qualitative pieces, syntheses of 
literature, historical analyses, and 
philosophical papers were not examined 
when addressing Research Questions 2, 3, 
and 4. Data were analyzed using the 
Statistical Package for the Social Sciences 
(SPSS version 14.0). Descriptive statistics, 
including frequencies, percentages, and 
crosstabs were used to summarize data. 

Results/F indings 

Question 1: What type of articles and how 
many of each type have been published in 
the Journal since 2000? 

Of the 251 pieces published in the 
Journal since 2000, 66% (n = 165) could be 


classified as exclusively quantitative in 
nature (Table 1). These included studies 
using either descriptive statistics only or 
both descriptive and inferential statistics. 
More articles using inferential statistics were 
published in the Journal than any other 
article type. In general, the number of 
studies using inferential statistics has 
increased, and, with the exception of 2005, 
the number of studies using only 
descriptive statistics has decreased since 
2000 . 

With the exception of 2005, in which the 
proportion of exclusively descriptive and 
descriptive and inferential studies published 
was about equal, the proportion of studies 
using inferential statistics has been greater 
than the proportion using descriptive 
statistics only since 2000. Although the 
majority of the studies published in 2000 
used only descriptive statistics, they were 
primarily census studies. 


Table 1 


Comparison of Types of Articles Published in the Journal of Agricultural Education Between 
2000 and 2006 (N = 251) 



2000 

2001 

2002 

2003 

2004 

2005 

2006 



Article type 

/ 

% 

/ 

% 

/ 

% 

/ 

% 

/ 

% 

/ 

% 

/ 

% 

Total 

% 

Descriptive 

26 

59.1 

8 

26.7 

8 

26.7 

7 

20.0 

7 

20.0 

13 

38.2 

8 

18.6 

77 

30.7 

Descriptive 
and inferential 

8 

18.2 

10 

33.3 

14 

46.7 

11 

31.4 

15 

42.9 

12 

35.3 

18 

41.9 

88 

35.1 

Qualitative 

4 

9.1 

7 

23.3 

4 

13.3 

6 

17.1 

4 

11.4 

1 

2.9 

7 

16.3 

33 

13.1 

Mixed 

methods 

4 

9.1 

2 

6.7 

2 

6.7 

2 

5.7 

3 

8.6 

3 

OO 

OO 

1 

2.3 

17 

6.8 

Delphi study 

0 

0 

0 

0 

0 

0 

5 

14.3 

1 

2.9 

3 

OO 

OO 

4 

9.3 

13 

5.2 

Synthesis of 
literature 

1 

2.3 

0 

0 

0 

0 

1 

2.9 

2 

5.7 

1 

2.9 

2 

4.7 

7 

2.8 

Historical 

research 

0 

0 

1 

3.3 

1 

3.3 

1 

2.9 

1 

2.9 

0 

0 

1 

2.3 

5 

2.0 

Philosophical 

0 

0 

1 

3.3 

0 

0 

1 

2.9 

1 

2.9 

0 

0 

1 

2.3 

4 

1.6 

Distinguished 

lecture 

1 

2.3 

1 

3.3 

1 

3.3 

1 

2.9 

1 

2.9 

1 

2.9 

1 

2.3 

7 

2.8 

Total 

44 

100 

30 

100 

30 

100 

35 

100 

35 

100 

34 

100 

43 

100 

251 

100 
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Question 2: To what extent have quantitative 
studies published in the Journal that use 
inferential statistics conveyed the practical 
significance of their research findings? 

Of the 165 quantitative studies published 
in the Journal since 2000, only those studies 
using inferential statistics (n = 88) were 
examined to answer Research Question 2. It 
should be noted that three of the studies 
using mixed methods used inferential 
statistics (including /-tests, analysis of 
covariance, and multiple regression). These 
three studies were not included in this 
analysis because they were not purely 
inferential. All other mixed methods studies 
used descriptive statistics only to summarize 
the data. 

Of the 88 inferential studies published in 
the Journal since 2000, 52% (n = 46) did 
not convey the practical significance of their 
research findings (Table 2). Although the 
manuscript submission guidelines for the 
Journal request that authors report effect 
sizes, a way to demonstrate practical 
significance of findings (Ary et ah, 2006; 
Miller, 1998), only 42 inferential studies did 
so. The most commonly reported form of 
practical significance was the R-squared 
tenn, followed by the squaring of the 
Pearson correlation coefficient r. 
Confidence intervals and standard error 
tenns were reported least often. Although 
the number of studies using inferential 
statistics generally increased between 2000 
and 2006, examining the reporting of 
practical significance over these 7 years 
revealed no related trends. Readers are 
encouraged to refer to Roberts and Dyer 


(2005) as an example of an article 
that accurately reported practical 
significance. 

Question 3: When necessary, to what extent 
have inferential studies reported in the 
Journal explicitly state research 
hypotheses? 

Of the 88 inferential studies published in 
the Journal since 2000, 74% (n = 65) did 
not explicitly state research hypotheses 
(Table 3). An examination of the data 
reveals that although the number of 
inferential studies that were published 
between 2000 and 2006 increased, fewer 
inferential studies explicitly stated research 
hypotheses over this time period. 

Question 4: To what extent have inferential 
studies reported in the Journal explicitly 
stated how the assumptions associated 
with statistical tests were addressed 
and confirmed? 

Of those same 88 inferential studies 
published in the Journal since 2000, nearly 
80% (n = 70) did not explicitly report how 
the assumptions associated with the 
statistical tests used were addressed and 
confirmed (Table 4). On examining the data, 
fewer than half the inferential studies 
published each year between 2000 and 2006 
stated outright how the assumptions 
associated with the inferential statistics used 
were accounted for. Readers are encouraged 
to refer to Pense and Leising (2004) as an 
example of how to report the procedures for 
addressing and confirming statistical 
assumptions. 


Table 2 


Number of Inferential Studies Published in the Journal that Reported Practical Significance of 
Research Findings (n = 88) 


Practical significance 
explicitly conveyed? 

2000 

2001 

2002 

2003 

2004 

2005 

2006 

Total 

Yes 

4 

3 

9 

2 

8 

6 

10 

42 

No 

4 

7 

5 

9 

7 

6 

8 

46 

Total inferential studies 

8 

10 

14 

11 

15 

12 

18 

88 
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Table 3 


Number of Inferential Studies Published in the Journal that Explicitly Stated Research 
Hypotheses (n = 88) 


Research hypotheses 
explicitly stated? 

2000 

2001 

2002 

2003 

2004 

2005 

2006 

Total 

Yes 

1 

1 

5 

3 

3 

4 

6 

23 

No 

7 

9 

9 

8 

12 

8 

12 

65 

Total inferential studies 

8 

10 

14 

11 

15 

12 

18 

88 

Table 4 

Number of Inferential Studies Published in the Journal Explicitly Stating How the Assumptions 
Associated with Statistical Tests Used Were Addressed and Confirmed (n = 88) 

Statistical assumptions 
explicitly addressed? 

2000 

2001 

2002 

2003 

2004 

2005 

2006 

Total 

Yes 

1 

0 

4 

1 

6 

2 

4 

18 

No 

7 

10 

10 

10 

9 

10 

14 

70 

Total inferential studies 

8 

10 

14 

11 

15 

12 

18 

88 


Discussion 

Research Question 1 examined the types 
of articles published in the Journal and the 
frequency of article types published since 
2000. In general, the proportion of studies 
using inferential statistics has been greater 
than the proportion using descriptive 
statistics only since 2000. However, in 2000, 
the majority of studies published in the 
Journal used descriptive statistics only, a 
value twice as high as the number of 
descriptive studies published in all other 
years. In describing the number and types of 
articles published in the Journal of 
Agricultural Education between 1990 and 
1999, Lindner, Murphy, and Briers (2001) 
found that about 44% of the articles 
reporting sampling procedures were census 
studies. In this study, the majority of studies 
published in 2000 were census studies and 
used descriptive statistics (describing 
parameters) because inferential statistics are 
not warranted when a census is attainable. 
Thus, 2001 represents a year of substantial 
change in the number of inferential studies 


being published in the Journal, a trend that 
has continued for over 5 years. 

Research Questions 2, 3, and 4 were 
developed based on suggestions of Miller 
and others and referred to the rigor with 
which studies in agricultural education have 
documented research methods and findings. 
Research Question 2 examined the extent 
that studies using inferential statistics 
conveyed the practical significance of 
research findings. Of those studies reporting 
practical significance, one of the most 
frequently reported forms of practical 
significance (over reporting the R-squared 
tenn in regression analysis) was squaring of 
the Pearson correlation coefficient. Miller 
(1994, 1998) advocated for more authors 
using relational statistics such as 
correlations to square the correlation 
coefficient to describe the amount of 
variability in the dependent variable which 
is explained by having knowledge of an 
independent variable. Perhaps this 
form of practical significance reporting 
was garnered following Miller’s 
suggestions. 
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Research Question 3 examined the 
extent to which inferential studies explicitly 
stated research hypotheses. Although Miller 
(1994, 1998) advocated that researchers in 
agricultural education outright state their 
hypotheses as part of the scientific method, 
space limitations in the Journal may prevent 
such reporting. As Dyer et al. (2003) 
suggested, current page limits may 
negatively affect the ability of authors to 
adequately address suggestions proposed to 
enhance the rigor of the discipline’s research 
reporting, such as the explicit reporting of 
hypotheses. Perhaps authors of inferential 
studies published in the Journal desire to 
report research hypotheses, thus adhering to 
the suggestions proposed by Miller and 
others, but are not given adequate space to 
do so. This study found that although the 
majority of inferential articles published in 
the Journal between 2000 and 2006 
provided an adequate review of literature 
and study rationale, many of which “hinted” 
at what was to be expected in the data, few 
studies stated outright the research 
hypotheses built from the associated 
literature review. 

Finally, Research Question 4 examined 
the extent to which inferential studies 
published in the Journal explicitly stated 
how the assumptions associated with 
statistical tests used were addressed and 
confirmed. Although the majority of 
inferential studies failed to discuss the 
assumptions upon which the statistical tests 
used were founded, as Dyer et al. (2003) 
discussed, space limitations in published 
manuscripts may be one reason for this. 
Miller (1994) argued that perhaps journal 
reviewers/editors will only accept articles 
that are “significant” or where something 
“significant” has been found. However, 
when writers omit stating how the 
assumptions behind the statistical tests they 
used were addressed and confirmed, might 
this imply shotgun empiricism — the 
unwarranted use of statistical tests in an 
attempt to find statistical significance — and, 


as Miller (1994) suggests, increase the 
likelihood of publication? 

Considering how one might develop a 
lesson plan on the steps to hanging a picture 
on a wall might be useful in conveying the 
importance of explicitly stating how 
statistical assumptions are dealt with. For 
example, a properly written lesson plan on 
hanging a framed picture would provide a 
list of materials needed to accomplish the 
task and a statement about the setting where 
the task should occur (assumptions about 
where the lesson plan is most relevant). One 
could use a large mallet to secure a rusty 
nail into the wall of a barn to hang a picture, 
but a more delicate, smaller hammer and 
cleaner nail would likely be more desirable 
for hanging a picture in a fine restaurant. 
Without a statement of assumptions 
regarding materials available and the most 
appropriate setting for learning to occur, a 
teacher might use the lesson plan in a way 
different from how its author intended it to 
be used and experience complications. 
Similarly, if inferential studies published in 
the Journal do not explicitly state how the 
assumptions associated with statistical tests 
were addressed, readers might use findings 
of such studies in a way different from how 
the author(s) intended them to be used. 

Conclusions/Recommendations 

Just more than 35% of the studies 
published in the Journal between 2000 and 
2006 used inferential statistics; the majority 
of these did not convey the practical 
significance of their findings, explicitly state 
research hypotheses, or document how the 
assumptions associated with the inferential 
statistics used were addressed. An 
examination of those 88 inferential studies 
revealed that only 6.8% (n = 6) answered 
Miller’s (1994, 1998, 2006) suggestions and 
conveyed practical significance, stated 
hypotheses, and addressed statistical 
assumptions in the same manuscript 
(Table 5). 
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Table 5 


Comparison of Reporting of Practical Significance, Research Hypotheses, and Statistical 
Assumptions in Inferential Studies Published in the Journal (n = 88) 


Research reporting done 

2000 

f 

2001 

f 

2002 

f 

2003 

f 

2004 

f 

2005 

f 

2006 

f 

Total 

% 

Neither P, H, or A 

4 

6 

3 

6 

4 

5 

7 

35 

39.8 

P only 

2 

3 

4 

1 

4 

3 

2 

19 

21.6 

H only 

0 

1 

1 

2 

1 

1 

1 

7 

8.0 

A only 

0 

0 

0 

1 

2 

0 

0 

3 

3.4 

P and H only 

1 

0 

2 

1 

0 

1 

4 

9 

10.2 

P and A only 

1 

0 

2 

0 

2 

0 

3 

8 

9.1 

H and A only 

0 

0 

1 

0 

0 

0 

0 

1 

1.1 

P, H, and A 

0 

0 

1 

0 

2 

2 

1 

6 

6.8 

Total 

8 

10 

14 

11 

15 

12 

18 

88 

100 


Note. P = Practical significance conveyed, H = Hypotheses stated, A = Assumptions regarding 
statistics addressed. 


In the same way educators are often 
encouraged to examine and follow 
exemplary lesson plans in an effort to 
enhance their teaching, the authors 
recommend that agricultural education 
researchers use inferential studies that 
conveyed practical significance, stated 
hypotheses, and addressed statistical 
assumptions as an example when developing 
inferential studies in the future. The authors 
randomly selected one of these exemplary 
studies published in the Journal and have 
reported it here: Pate, Wardlow, and 
Johnson (2004). 

In considering the suggestions proposed 
by Miller and the findings of this study, the 
authors make the following additional 
rec ommendations : 

1. As Dyer et al. (2003) suggested, an 
increase in the number of pages 
allotted to each article would provide 
authors of inferential studies with 
additional space to report research 


hypotheses and discuss how 
statistical assumptions were 
addressed. 

2. When adequate time and resources 

are unavailable to conduct census 
studies, researchers should consider 
gathering a smaller, representative 

sample and using inferential statistics 
to make predictions about the 

population of interest. 

3. To improve the image of agricultural 

education research, described earlier 
as being of lacking quality (Buriak & 
Shinn, 1989; Dyer et ah, 2003), 
authors of inferential studies in 
agricultural education are 
encouraged to explicitly state 
research hypotheses, explicitly 

address assumptions of the statistical 
tests they use, and consider the 
practical versus statistical 

significance of research findings. 
Individuals who teach statistics/ 
research methods courses and/or 
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advise agricultural education 
graduate students should encourage 
the same of their students. 
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